Papers with sentence representation learning
Simple Temperature Cool-down in Contrastive Framework for Unsupervised Sentence Representation Learning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies have focused on the effectiveness of contrastive learning in deep learning. |
| Approach: | They propose a method to improve sentence representation of unsupervised contrastive learning by examining the role of temperature in VRL and SRL. |
| Outcome: | The proposed method improves representation of unsupervised contrastive learning by cooling the temperature of the representation space. |
Bootstrap Your Own PLM: Boosting Semantic Features of PLMs for Unsuperivsed Contrastive Learning (2024.findings-eacl)
Copied to clipboard
| Challenge: | Existing studies have shown that SimCSE significantly improves the performance of pretrained language models on the sentence representation benchmark. |
| Approach: | They propose a method called IFM which reduces the tendency of contrastive models for VRL to rely on feature-suppressing shortcut solutions. |
| Outcome: | The proposed method reduces the tendency of contrastive models for VRL to rely on feature-suppressing shortcut solutions. |
Hyper-CL: Conditioning Sentence Representations with Hypernetworks (2024.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to sentence embeddings do not capture fine-grained semantics of sentences. |
| Approach: | They propose a method that integrates hypernetworks with contrastive learning to generate conditioned sentence representations. |
| Outcome: | The proposed method narrows the performance gap with the bi-encoder architecture while maintaining the time efficiency characteristic of the tri-encoding approach. |
A Comprehensive Survey of Sentence Representations: From the BERT Epoch to the CHATGPT Era and Beyond (2024.eacl-long)
Copied to clipboard
Abhinav Ramesh Kashyap, Thanh-Tung Nguyen, Viktor Schlegel, Stefan Winkler, See-Kiong Ng, Soujanya Poria
| Challenge: | Sentence representations are a critical component in NLP applications such as retrieval, question answering, and text classification. |
| Approach: | They present a systematic review of the literature on sentence representations focusing mostly on deep learning models. |
| Outcome: | The proposed methods highlight the key contributions and challenges in this area and suggest potential avenues for improving the quality and efficiency of sentence representations. |
Self-Guided Contrastive Learning for BERT Sentence Representations (2021.acl-long)
Copied to clipboard
| Challenge: | Existing methods to derive sentence embeddings from pre-trained Transformers are unclear . a self-guided training method is used to fine-tune BERT in a supervised fashion . |
| Approach: | They propose a contrastive learning method that utilizes self-guidance to improve BERT sentence representations. |
| Outcome: | The proposed method is more effective than baselines on diverse sentence-related tasks and robust to domain shifts. |
Sentence Representation Learning with Generative Objective rather than Contrastive Objective (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing sentences-level training objectives focus on acquiring sentence-level representations, but they lack effective self-supervised objectives. |
| Approach: | They propose a generative self-supervised learning objective based on phrase reconstruction to improve sentence representation. |
| Outcome: | Empirical results show that the proposed objective outperforms current methods on STS benchmarks and retrieval and reranking tasks. |
A Simple Angle-based Approach for Contrastive Learning of Unsupervised Sentence Representation (2024.findings-emnlp)
Copied to clipboard
| Challenge: | a promising baseline SimCSE has made notable breakthroughs in unsupervised SRL . however, there is still room for designing a novel contrastive framework specifically targeted for SRL. |
| Approach: | They propose an angle-based similarity function for a contrastive objective and propose a new approach for SRL. |
| Outcome: | The proposed approach shows better training dynamics on SRL than the standard cosine similarity function. |
Mining Discourse Markers for Unsupervised Sentence Representation Learning (N19-1)
Copied to clipboard
| Challenge: | Current state of the art systems in NLP heavily rely on manually annotated datasets, which are expensive to obtain and are ineffective to extract. |
| Approach: | They propose to automatically discover sentence pairs with relevant discourse markers and apply it to massive amounts of data. |
| Outcome: | The proposed method can learn transferable sentence embeddings from 174 discourse markers even for rare markers such as “coincidentally” or “amazingly”. |
Large Language Models can Contrastively Refine their Generation for Better Sentence Representation Learning (2024.naacl-long)
Copied to clipboard
| Challenge: | Existing methods for training contrastive learning based sentence embedding models are largely influenced by the quality of sentence pairs. |
| Approach: | They propose a framework that decomposes LLMs into three stages for training . they propose to refine the generated content at these stages to ensure only high-quality sentence pairs are utilized to train a base contrastive learning model. |
| Outcome: | The proposed framework surpasses ChatGPT and ChatGPP in terms of performance. |
Pairwise Supervised Contrastive Learning of Sentence Representations (2021.emnlp-main)
Copied to clipboard
| Challenge: | Recent efforts to improve sentence representation learning have a common weakness . siamese or triplet loss only learns from individual sentence pairs or tripletes . |
| Approach: | They propose a discrimination-based approach to bridge entailment and contradiction understanding with categorical concept encoding. |
| Outcome: | The proposed method outperforms the state-of-the-art method on downstream tasks . it improves 10%–13% on clustering tasks and 5%–6% on STS tasks compared with the previous method . |
Differentiable Data Augmentation for Contrastive Sentence Representation Learning (2022.emnlp-main)
Copied to clipboard
| Challenge: | a contrastive learning framework is used to fine-tune pre-trained language models with unlabeled sentences or labeled sentences. |
| Approach: | They propose a method that makes hard positives from unlabeled sentences . they use a prefix attached to a model to allow for differentiable data augmentation . |
| Outcome: | The proposed method yields significant improvements over existing methods under semi-supervised and supervised settings. |
Transfer Fine-Tuning: A BERT Case Study (D19-1)
Copied to clipboard
| Challenge: | Recent advances in sentence representation learning have improved the performance of BERT models, but the computational power required is an obstacle preventing practical applications from adopting the technology. |
| Approach: | They propose to inject phrasal paraphrase relations into BERT to generate suitable representations for semantic equivalence assessment instead of increasing model size. |
| Outcome: | The proposed model improves a smaller model while maintaining the model size. |
Clustering-Aware Negative Sampling for Unsupervised Sentence Representation (2023.findings-acl)
Copied to clipboard
| Challenge: | Using clustering-aware learning, in-batch negatives are often ignored in sentence representation learning. |
| Approach: | They propose a method that integrates cluster information into contrastive learning for unsupervised sentence representation learning. |
| Outcome: | The proposed method compares favorably with baselines on semantic textual similarity tasks. |
Generate, Discriminate and Contrast: A Semi-Supervised Sentence Representation Learning Framework (2022.emnlp-main)
Copied to clipboard
| Challenge: | Existing supervised sentence embedding techniques rely on expensive human-annotated sentence pairs as the supervised signals. |
| Approach: | They propose a semi-supervised sentence embedding framework that leverages large-scale unlabeled data. |
| Outcome: | The proposed framework surpasses state-of-the-art methods on four domain adaptation tasks. |
DATA-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning (2024.findings-acl)
Copied to clipboard
| Challenge: | Existing methods to improve sentence representation learning (SRL) ignore the potential interference problems across tasks and instances. |
| Approach: | They propose a multi-task instruction tuning method that arranges the order of multi- task data for training to minimize interference risks. |
| Outcome: | The proposed method can boost the performance of state-of-the-art methods. |